Benchmark：

Settings

max_query_memory_usage 2147483648
max_threads 4
query_out_of_memory_behavior spilling
sort_spilling_batch_bytes  1048576

DataSet: tpch sf100

SQL

select * from orders order by o_totalprice desc ignore_result;

Analysis:
In this test, there are only two operators that use a lot of memory, one is sort and the other is scan.
The sort collect step eats up all the available memory within the limit, while scan has no memory limit logic, so if scan allocates a lot of memory at this time, for example, if there are many columns and the columns are relatively large, the limit will be exceeded.

Currently, the sort restore step does not read the global memory usage, but instead statically determines the available memory by referring to the memory usage of the collect step, which is an inelastic strategy. After the restore step, since there is no more scanning, we can't see any problem.

Bug fixed:
If the block coming from upstream is large, it is possible to directly spill it without sorting, which will cause huge deviation in memory estimation of the restore step. This bug can be made more obvious by making sort_spilling_batch_bytes smaller.
The current tuning should ensure that only the first spill in the collect step exceeds the limit.

It is recommended to use a larger sort_spilling_batch_bytes, because the smaller the sort_spilling_batch_bytes, the larger the num_merge, the more fragmented the blocks, and the larger the num_merge, the more destabilizing the elements will be. (More testing is needed on what the destabilizing elements are)

Two issues with this PR:

1. conflicts Cargo.lock
2. test_logs always failed(🤔 )
3. explain CI failed: https://github.com/databendlabs/databend/actions/runs/16924936615/job/47961073098?pr=18520

Docker Image for PR

• tag: pr-18520-fdc90d3-1755051739

note: this image tag is only available for internal use.

Docker Image for PR

• tag: pr-18520-5b6746a-1755083362

note: this image tag is only available for internal use.